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Dear  Susan; 

This  letter  is  a  quarterly  progress  report  for  the  On-Line  Assessment  of  Expertise  project 
(Grant  N0001 4-91 -J-1 532).  It  covers  the  period  from  January,  1992  to  March,  1992. 

The  major  accomplishment  during  the  report  period  was  designing  3  studies  and  running  one  of 
them. 

The  initial  study  was  designed  to  test  the  user  interface  for  gathering  data  about  problem 
solving,  test  our  choice  of  problems,  and  gather  verbal  protocols  that  could  be  compared  to  the 
user-interface  action  sequences  collected  by  the  system.  Ten  subjects  were  drawn  from  the  Pitt 
intro  physics  class  for  science  and  engineering  majors.  Each  subject  was  first  introduced  to 
the  system  and  taught  how  to  use  the  editors  to  solve  problems.  They  were  also  shown  how  to 
call  up  any  of  the  ten  example  from  their  textbook  and  read  it  via  the  poor-man’s  eyetracker. 
They  then  solved  problems  relevant  to  the  target  chapter  (chapter  5  --  Newton's  laws  applied 
to  translational  motion  without  friction)  but  taken  from  other  textbooks.  They  worked  for  two 
hours,  solving  as  many  problems  as  they  could.  They  were  asked  to  give  a  verbal  protocol  as 
they  worked,  and  prompted  when  they  fell  silent.  The  first  five  subjects  uncovered  significant 
problems  with  the  user  interface,  so  their  runs  were  truncated  and/or  protocols  were  not 
taken.  They  helped  us  improve  the  interface  to  the  point  where  subjects  now  find  it  simple  to 
learn  and  use.  but  their  data  has  been  discarded.  Subsequent  analysis  will  be  based  on  the  five 
remaining  subjects.  , 

The  second  study  will  begin  running  this  week.  The  second  study  is  designed  to  test  the  user 
interfaces  for  various  kinds  of  qualitative  problem  solving:  The  problem  classification  task, 
stating  a  basic  approach,  estimating  difficulty  and  McClosky-style  conceptual  problem  solving. 
We  plan  on  running  ten  subjects,  and  expect  the  first  five  to  be  used  only  as  pilots  to  debug  the 
interfaces. 

The  third  study  will  be  conducted  during  the  Pitt  summer  session.  The  timing  is  crucial.  We 
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plan  to  catch  physics  students  just  before  they  are  ready  to  read  Chapter  5,  then  have  them 
make  their  first  reading  of  the  chapter  using  Olae’s  poor-man's  eyetracker  to  read  the 
examples.  We  hope  to  detect  self-explanation  from  the  latency  data.  We  will  collect  verbal 
protocols  on  half  the  problems,  and  score  these  in  the  usual  way  in  order  to  determine  where  the 
student  is  self-explaining.  The  latency  data  will,  we  hope,  correlate  strongly  with  the  verbal 
data.  Moreover,  the  distribution  of  latencies  should  be  similar  on  the  problems  were  verbal 
protocols  were  not  given. 

Now  that  we  have  some  problem  solving  data,  we  have  begun  developing  a  technique  for  matching 
the  actions  gather  from  the  user  interface  to  the  lines  of  reasoning  generated  by  Cascade  and 
embedded  in  the  Bayesian  networks.  This  appears  to  be  a  very  difficult  problem,  and  may 
require  some  significant  representational  changes  in  Cascade  (currently,  it  does  not  distinguish 
mental  actions  from  physical  actions.  If  it  did,  then  we  would  have  a  much  easier  time  matching 
its  predictions  to  the  data,  which  consist  of  physical  actions  only). 

As  part  of  a  formative  evaluation  of  Olae,  we  plan  on  comparing  our  interpretation  of  the  verbal 
protocols  to  the  interpretation  of  the  action  protocols  derived  by  Olae.  However,  we  do  not  have 
the  money  to  hire  a  transcriptionist  for  the  verbal  protocols,  nor  the  patience  to  do  it 
ourselves,  so  we  have  begun  to  looking  into  hooking  up  some  digital  audio  technology  to  Olae  so 
that  when  we  replay  the  action  sequences,  it  will  also  replay  the  verbal  protocol  that  occurred 
in  the  vicinity  of  the  current  action.  We  are  aware  of  the  poor  track  record  of  attempts  to  work 
around  the  transcription  bottleneck,  but  there  is  this  new  digital  speech  compression  technology 
that  could  potentially  be  used,  so  we  feel  that  it  is  worth  a  few  days  work  to  evaluate  the 
feasibility.  It  could  save  us  a  great  deal  of  time  in  the  long  run  if  it  works. 

In  January,  two  graduate  students  associated  with  the  project  began  working  on  individual 
projects  aimed  at  building  models  to  help  us  interpret  the  qualitative  data.  Jonathan  Rubin  is 
building  a  model  of  solution  planning  that  we  hope  will  allow  us  to  model  three  tasks:  the 
classification  task,  the  stating  of  a  basic  approach  to  a  quantitative  problem,  and  the  estimation 
of  a  quantitative  problem's  difficulty.  Progress  during  the  first  two  months  was  unexpectedly 
rapid.  Jon  found  that  some  small  changes  to  Cascade  sufficed  to  have  it  create  a  solution  plan  and 
run  it.  During  the  last  month,  however,  things  have  slowed  down  as  we  try  to  debug  the  system 
and  get  it  to  run  on  more  problems.  . 

Sigalit  Ur  is  doing  a  master  thesis  (which  is  required  of  all  our  PhD  students)  on  conceptual 
problem  solving  and  conceptual  change.  She  is  building  on  top  of  work  done  by  Rolf  Ploetzner,  a 
graduate  student  who  visited  my  group  last  year  and  wrote  an  initial  version  of  a  conceptual 
problem  solver  based  on  Cascade.  Sigalit  began  by  collecting  protocols  from  4  subjects  as  they 
solved  conceptual  problems.  She  found  the  usual  menagerie  of  misconceptions,  and  got  some 
ideas  of  the  chronology  of  the  process.  In  particular,  she  found  subjects  would  often  produce 
their  answer  quickly,  but  when  pressed  to  justify  it,  would  come  up  with  all  kinds  of  wonderful 
recollections  (cases),  imagery  and  p-prim-filled  reasoning.  Sigalit  and  I  have  had  an 
intensive  series  of  meetings  with  Micki  Chi  and  her  student,  Jim  Slotta,  who  are  working  on 
conceptual  shifts  in  physics  and  biology.  We  have  also  been  corresponding  with  Andy  di  Sessa 
and  Jeremy  Roschelle  over  the  net.  We're  trying  to  bring  all  these  various  intuitions  and 
empirical  generalizations  together  and  to  operationalize  them  in  a  conceptual  problem  solver. 
This  should  help  us  understand  how  to  interpret  the  data  from  Olae's  conceptual  problem  solving 
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task  as  well  as  setting  the  stage  for  further  work  on  models  of  conceptual  change. 


Copies: 

Administrative  Grants  Officer 
Office  of  Naval  Research 
Resident  Representative  N66005 
Administrative  Contracting  Officer 
The  Ohio  State  Univ.  Research  Center 
1314  Kinnear  Rd. 

Columbus,  1  43212-1194 


Sincerely, 


Kurt  A.  VanLehn 

Associate  Prof.  Computer  Science 
Senior  Scientist,  LRDC 


Director,  Naval  Research  Laboratory 
Attn:  Code  2627 
Washington,  DC  20375 

Defense  Technical  Information  Center 
Building  5,  Cameron  Station 
Alexandria,  VA  22304-6145 


Statement  A  per  telecon 

Dr.  Susan  Chipman  ONR/Code  1142 

Arlington,  VA  22217-5000 
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